Apparatus and method for photogrammetric surgical localization

ABSTRACT

A method and apparatus for defining the location of a medical instrument relative to features of a medical workspace including a patient&#39;s body region are described. Pairs of two-dimensional images are obtained, preferably by means of two video cameras making images of the workspace along different sightlines which intersect. A fiducial structure is positioned in the workspace for defining a three dimensional coordinate framework, and a calibration image pair is made. The calibration image pair comprises two 2D projections from different locations of the fiducial structure. After the calibration image pair is made, the fiducial structure is removed. A standard projection algorithm is used to reconstruct the 3D framework of the fiducial structure from the calibration image pair. Appropriate image pairs can then be used to locate and track any other feature such as a medical instrument, in the workspace, so long as the cameras remain fixed in their positions relative to the workspace. The computations are desirably performed with a computer workstation including computer graphics capability, image processing capability, and providing a real-time display of the workspace as imaged by the video cameras. Also, the 3D framework of the workspace can be aligned with the 3D framework of any selected volume scan, such as MRI, CT, or PET, so that the instrument can be localized and guided to a chosen feature. No guidance arc or other apparatus need be affixed to the patient to accomplish the tracking and guiding operations.

This is a continuation of application Ser. No. 09/173,138 filed on Oct.15, 1998, now pending, which is a continuation of application Ser. No.08/801,662 filed on Feb. 18, 1997, now U.S. Pat. No. 5,836,954, which isa continuation of application Ser. No. 08/145,777, filed on Oct. 29,1993, now U.S. Pat. No. 5,603,318, which is a continuation-in-part ofapplication Ser. No. 07/871,382, filed on Apr. 21, 1992, now U.S. Pat.No. 5,389,101.

BACKGROUND OF THE INVENTION

1. Field

The application is related to techniques for mapping internal structuresin the body of an animal or human, and more particularly to suchtechnique for localizing a medical instrument with respect to anatomicalfeatures or the like during surgical or other medical procedures.

2. State of the Art

Various scanning apparatus and methods are known for imaging and mappingbody structures, which provide target location data for surgical andother medical procedures. One group of methods, including stillphotography, videography, radiological x-rays, and angiography,typically produces only a two-dimensional projection of athree-dimensional object. For purposes of this application, this firstgroup will be termed "two-dimensional" or "2D" imaging.

A second group of methods, of which computerized tomographic (CT)scanning, positron emission tomography (PET) scans, and magneticresonance (MRI) imaging are exemplary, provides three-dimensional(abbrev. "3D" herein) information about internal structures (i.e.,structures not visible from the exterior of the patient). Thethree-dimensional information about the internal volume is reconstructedfrom multiple scans of a known thickness (generally about a millimeter)made along parallel planes displaced from each other by a knowndistance, usually of the order of millimeters. An example of such areconstructed volume image is depicted in FIG. 1A, including thecontours of a selected anatomical feature within the brain. In thisapplication, methods in this second group will be referred to as"volume" scanning or imaging.

In performing resection or other surgical manipulations, it is highlydesirable to correlate the location of instruments, patient anatomicalfeatures, or other elements or structures placed in the surgical field,and generally as seen by the surgeon, with the location of internaltargets or features as visualized by one of the volume scanningtechniques. Such a correlation process is often termed "localization".

A commercially available device for localization in neurosurgery is theBrown-Roberts-Wells (abbrev. BRW) localizer (U.S. Pat. Nos. 4,341,220,and 4,608,977). The BRW system includes a large ring-like structurewhich surrounds the patient's head and is fixed in place. The ringestablishes a 3D coordinate system with respect to the patient's head. Aseparate calibration unit having an array of rod elements is fixed tothe ring to surround the head during the production of volume scanand/or 2D images. The rod elements have known coordinates in the 3Dcoordinate system established by the ring, and produce spots in thevolume scans. Other features in the volume scans can then be assignedcoordinates in the 3D coordinate system established by the ring, bycorrelation with the known coordinates of the rod elements producing thespots.

After the images are made, the calibration unit is detached from thering, and a guidance arc calibrated to the 3D coordinate system of thering is attached in its place. The guidance arc provides coordinatereference information which may be used to guide a medical instrument.The medical instrument is usually attached to the guidance arc.

The BRW system has several disadvantages. The ring is cumbersome anduncomfortable for the patient, but it must be affixed in place when thevolume and/or 2D scans are made, and kept there until the medicalprocedure is complete. It is possible to remove the ring after the scansare made, but precise repositioning is critical to avoid error inlocalization. Accurate repositioning is difficult, so present practicegenerally is to keep the ring in place until after the surgery. When notattached to the guidance arc, the position of a medical instrument interms of the 3D coordinate system of the ring, and therefore in respectto the features identifiable in the volume or 2D scan, is not accuratelyknown.

U.S. Pat. No. 4,618,978 to Cosman discloses a localizer device for usewith a BRW-type system, including an open box composed of connectedrods, which surrounds the patient's head and constitutes a calibrationunit.

Alternatively, cranial implants of radio-opaque or MRI-opaque materialscan be made. Generally, a minimum of three implants are required forestablishing a three-dimensional space in volume scans. At present thismethod is considered very undesirable, in part because of the risk ofinfection or other complications of the implants.

Accordingly, a need remains for rapid, reliable, and inexpensive meansfor localizing a medical instrument relative to points of interestincluding both visible anatomical features and internal features imagedby volume and/or 2D methods. A need further remains for such means whichdoes not require the physical attachment of a reference unit such as theBRW ring to the patient. Highly desirably, such means would be useful totrack the position of a medical instrument in real time, and withoutrequiring that the instrument be physically attached to a referenceguide.

OTHER TERMS AND DEFINITIONS

A coordinate system may be thought of as a way to assign a unit set ofnumerical identifiers to each point or object in a selected space. TheCartesian coordinate system is one of the best known and will be used inthis paragraph by way of example. In the Cartesian coordinate system,three directions x, y, z are specified, each corresponding to one of thethree dimensions of what is commonly termed 3D (three-dimensional) space(FIG. 1B). In the Cartesian system, any point can be identified by a setof three values, x, y, z. The x, y and z directions can be said toestablish a "three-dimensional framework" or "coordinate framework" inspace. A selected point "A" can be described in terms of its valuesx_(a), y_(a), z_(z) ; these values specify only the location of point A.A different point B will have a different set of values x_(b), y_(b),z_(b). Such a set of values x,y,z for any selected point is referred toherein as the "coordinates" or "locational coordinates" of that point.When the position of a feature larger than a single point is beingdescribed, these terms are also understood to refer to a plurality ofsets of x,y,z values. Other types of coordinate systems are known, forexample spherical coordinate systems, and the terms "coordinates" and"locational coordinates" should further be understood to apply to anyset of values required to uniquely specify a point in space in a givencoordinate system.

The term "fiducial" is used herein as generally understood inengineering or surveying, to describe a point or marking, or a line,which is sufficiently precisely defined to serve as a standard or basisreference for other measurements.

SUMMARY OF THE INVENTION

The invention comprises apparatus and a method for defining the locationof a medical instrument relative to elements in a medical workspaceincluding a patient's body region, especially (but not limited to)elements seen by the surgeon. The apparatus develops a calibrated 3dimensional framework of the workspace from a pair of 2D images madefrom different fixed locations, and aligns the workspace framework witha 3D scan framework defined by a volume scan. A pair of video cameras isthe present preferred imaging means for obtaining the 2D image pairs.The apparatus is then operable to locate and track the position of amedical instrument during a medical procedure, with respect to featuresobservable in either the workspace images or in the volume scan. Apictural display of such location and tracking information is providedto aid a medical practitioner performing the procedure.

In a further embodiment, the computing means is operable toautomatically recognize and track the position of selected medical orsurgical instruments during a procedure, from the workspace images.

The apparatus may be described as follows. Workspace imaging means areprovided and positioned for producing a plurality of pairs of2-dimensional images of a medical workspace. Each image pair comprisestwo such images made in effect simultaneously along respective differentsightlines which intersect at an angle. Digitizing means are operablydisposed for digitizing each image to produce corresponding sets ofdigital output signals, one set for each image.

Calibration means are removably positionable in the workspace forcalibrating the workspace in terms of a three-dimensional coordinateframework. The 3D workspace framework is derived by computation from thetwo 2D projections of an image pair made with the calibration meanspositioned in the workspace. The calibration means comprises a step ofat least six fiducial points connected by a frame means consisting of aframe constructed to hold the fiducial points in fixed spatial relationto each other. Although a calibration means with a set of at least sixfiducial points is preferred, it is understood that the calibrationmeans only requires a sufficient number of fiducial points to derive the3D workspace framework. The frame need not include any means forattaching the fiducial points to a patient. The set of fiducial pointshas known spatial parameters which define an arbitrary Cartesian3-dimensional coordinate system. These spatial parameters include 3Dlocation coordinates of each of the fiducial points. Optionally butdesirably, at least some of the actual distances between fiducial pointsshould be known, to calibrate the workspace in terms of a suitabledistance unit such as millimeters.

A computing means is connected to receive the digital output signalsreflective of the images. The computing means also has data input meansfor receiving scan data from a volume scan of the patient's body region.The scan data define a scan 3D coordinate framework and internalanatomical structures therein. The computing means is furtherconstructed or programmed to perform the following steps: 1) establish aworkspace coordinate framework in three dimensions from an image pairmade with said fiducial structure positioned within the workspace; 2)determine the locational coordinates in the workspace framework of anyselected point which can be identified from both images of said pair; 3)correlate the scan locational coordinates for each of three or moreselected landmarks observable in the scan with the workspace locationalcoordinates of the same landmarks as derived from a video image pair; 4)use the correlation of the workspace coordinates and the scancoordinates of the landmarks, to derive a transformation algorithm formapping selected other features from either the scan framework to theworkspace framework, or the converse; and 5) provide display signalsencoding a display reflective of one or both of the workspace imagesand/or a volume scan, as selected by a user. Display means are providedfor displaying the images encoded by the display signals.

Optionally but highly desirable, the computing means has computergraphics capability for producing graphic icons overlaid upon thedisplayed images. Such icons include a cursor which the user employs toselect features in the displayed images for computation of theircoordinates or other operations.

A method of surgical guidance may be described as follows. First, afiducial structure having six or more fiducial points defining twodistinct, non-orthogonal planes is positioned in a medical workspace.Workspace imaging means are disposed for making pairs of two-dimensionalimages of the workspace in which the two member images are made alongdifferent but intersecting sightlines. A calibration image paircomprising images of the workspace with the fiducial structure is made.The fiducial structure is removed from the workspace.

A projection algorithm is applied to reconstruct a workspace 3Dcoordinate framework from the calibration image pair. At least oneadditional 3D scan framework is obtained from a corresponding volumescan of the patient's body region. At least three landmarks identifiablein both the volume scan and the workspace image pair are selected, andthe coordinates for the three landmarks are determined in both theworkspace framework and the scan framework. From these determinedcoordinates, a process is developed for aligning the scan framework withthe workspace framework, and transformation algorithms for convertingcoordinates from one of the frameworks to the other are computed.

A target of interest in the volume scan is identified, and its scancoordinates are determined and converted to workspace coordinates. Afeature of interest in the workspace, such as a fiducial mark on ascalpel, is identified. The workspace coordinates of the fiducial markand of the scalpel tip (whose distance from the fiducial mark is known),plus a vector describing the direction of the scalpel, are determined.Optionally but highly desirably, both the target and the scalpelincluding the scalpel tip position are displayed in an image of theworkspace. The path of the scalpel tip is extrapolated along the vectorfor a distance sufficient to determine whether the tip will reach thetarget on this path. If not, the direction of the scalpel is adjustedand the process of localizing the tip and extrapolating its path isrepeated until the extrapolated path is deemed adequate by a user,and/or until the medical procedure is complete.

The invention also includes a fiducial structure for establishing athree-dimensional coordinate framework for a photographic image pair,comprising a sufficient umber of fiducial indicators arranged to athree-dimensional coordinate system and frame means for supporting andconnecting the fiducial indicators, wherein the frame means includes aframe constructed to hold the fiducial indicators in fixed relation toeach other.

BRIEF DESCRIPTION OF THE DRAWINGS

In the figures, which illustrate what is presently regarded as the bestmode for carrying out the invention, like reference numbers indicatelike elements of the apparatus:

FIG. 1A is a cartoon of a volume scan of a patient's head;

FIG. 1B depicts a 3-dimensional coordinate system;

FIG. 2 is a block diagram depicting the basic elements of a videolocalization system of the invention;

FIG. 3 depicts an embodiment of the fiducial structure in greaterdetail;

FIG. 4 depicts a pair of images made from different positions of asurgical workspace including a patient's head, with the fiducialstructure of the invention positioned for calibrating the workspace;

FIG. 5 is a flow chart of a portion of the operation of a furtherembodiment in which the control means is configured to provide objectrecognition and location of medical instruments and the like in theimage field;

FIG. 6 shows a top plan view of a medical instrument.

DETAILED DESCRIPTION OF THE ILLUSTRATED EMBODIMENT

FIG. 2 is a block diagram depicting the basic elements of a workingembodiment of a video localization system of the invention. A pair ofvideo cameras 200, 202 are positioned for making a pair of images alongrespective sightlines 204, 206, of a medical workspace 208 whichincludes a patient's body region here shown to be the patient's head210. Cameras 200, 202 are arranged to have an angle 212 betweensightlines 204, 206, such that both cameras image the workspace 208.Workspace 208 is effectively defined by the overlapping fields of viewof the respective images made by cameras 202, 202. Angle 212 ispreferably between about 30° and 150°. However, any angle greater thanzero degrees and not equal to 180° can be used.

Alternatively, cameras 200, 202 may be replaced by a single camera whichis moved back and forth between first and second positions to takeimages along respective sightlines 204, 206. In the latter case, it isimportant that the camera be precisely positioned in the first andsecond positions when making the respective images of an image pair.Positioning means may be provided for establishing fixed attachmentpoints for attachment of a camera, to facilitate such repositioning.Whether one camera or two cameras are used, what is significant is thatthe system takes pairs of images of workspace 208, each member of animage pair made along different sightlines intersecting in workspace208.

A fiducial structure 220 (described in greater detail with reference toFIG. 3) is shown positioned in the workspace 208 proximal to the head210. During use, fiducial structure 220 can be held in position by anysuitable support means (not shown). One suitable support means would bea bar with a clamp arm attached to a ring stand or the like. Notably,fiducial structure 220 is neither affixed to, nor in contact with, thepatient's head 210. Fiducial structure 220 may be removed from workspace208 when it is not required.

Cameras 200, 202 are communicatively connected to image digitizing means230, which produces two sets of digitized image signals, eachrepresentative of a respective image detected by one of the two cameras.Digitizing means 230 is in turn connected to send the digitized imagesignals to computing means 232.

Computing means 232 receives the digitized image signals from digitizingmeans 230 and is operable in response to compute display signalsrepresentative of the digitized video image(s) of workspace 208 as seenby one or both of cameras 200, 202. Computing means 232 comprises atleast a central processing unit, memory means which includes bothvolatile and nonvolatile memory components, data input means, an imageprocessing/computer graphics subunit, and output means for outputtingdisplay signals. The foregoing components of computing means 232 arefunctionally interconnected generally as known in the art of computing.In a further embodiment, computer means 232 is operable to combineimages of the workspace made from each of the two different positions toproduce a single stereo image.

Computing means 232 supplies the display signals to a display unit 240which may be a video display, a CRT monitor, or the like. Display unit240 converts the display signals to a video image of the workspace 208as seen by either or both of cameras 200, 202. Display unit 240 ispositioned for reading viewing by medical personnel performingprocedures in the workspace. Preferably, display unit 240 is constructedto provide sufficient resolution to adequately distinguish significantcomponents in images of the workspace 208. In FIG. 2, display unit 240is depicted as having a single viewing screen showing the image as seenby camera 200. This embodiment is provided with a single screen fordisplaying visual depictions of the available scans and images. Thesemay includes the image made by camera 200, the image made by camera 202,scan images derived from volume scanning methods, X-ray images made byX-rays and including angiograms, etc., as selected by a user operatingcomputing means 232. The user may switch the display from one to anotherof the various visual depictions, as desired. Also, one or more featuresof a first selected depiction, or the entire first selected depiction,can be overlaid on a second selected view.

Alternatively, display unit 240 may contain a plurality of viewingscreens arranged for simultaneously displaying in separate screens, theselected depictions.

Computing means 232 also provides graphic display signals to the displayunit 240, to produce graphic icons overlaid upon the selected displayedimage. The graphic icons should include a cursor which can be positionedby the user at a feature of interest in the displayed image.

Computing means 232 is further constructed, or alternatively programmed,to compute a workspace coordinate framework which defines workspace 208in terms of three-dimensional Cartesian coordinates in useful distanceunits, for example millimeters. The workspace coordinate framework iscomputer from the two digitized 2-dimensional images of the fiducialstructure 220 provided respectively by cameras 200, 202, plus the knownlocation parameters of fiducial points on fiducial structure 220(described in more detail in reference to FIG. 3). In the workingembodiment, computing means 232 performs these computations according toa well-known projection algorithm, originally developed by Bopp andKrauss (Bopp, H., Krauss, H., An orientation and calibration method fornon-topographic applications, Photogrammetric Engineering and RemoteSensing, Vol. 44, Nr. 9, September 1978, pp. 1191-1196).

The memory means of computing means 232 is constructed or programmed tocontain the known location parameters of fiducial structure 220, whichare required for performance of the computations producing the workspacecoordinate framework from the two 2D images. In the working embodiment,these known location parameters include three-dimensional Cartesiancoordinates for each of the fiducial points and the actual distancesbetween some of the fiducial points as measured from fiducial structure220. The latter distances are not required for establishing theworkspace framework, but are used to calibrate the framework in terms ofuseful real distance units.

Once the workspace coordinate framework has been computed, computingmeans 232 is further operable to compute the 3D location coordinateswithin the workspace framework of any feature of interest whose positionmay be observed by means of the images made with both of cameras 200,202. Such workspace location coordinates will be accurate provided thetwo images are made from substantially the same positions relative toworkspace 208 as during the establishment of the three-dimensionalframework with the fiducial structure.

Features which can be observed by means of the images made by cameras200, 202 include both features actually seen in both images, andfeatures which are not within the field of view of one or both imagesbut whose position can be indicated by use of a pointer with at leasttwo fiducial marks, where the distance between at least one of thefiducial marks and the tip of the pointer is known. Two fiducial marksare needed to establish the direction with respect to the workspacecoordinate framework, of a vector representing the linear direction ofthe pointer. Alternatively, any other marker(s) useful to compute thevector direction may be employed.

Examples of features of interest include externally-placed portions ofscan markers used for volume and/or 2D scans, anatomical features on orwithin the patient including skull surface contours, marks on thepatient's skin, medical instruments and devices, etc.

Computing means 232 further has data input means 238 for receiving datarepresenting one or more scans produced by volume imaging methods (PET,MRI, CT) and/or by 2D imaging methods (X-rays, angiograms) etc. In analternate embodiment, computing means digitizes the CT and/or MRI volumescans and integrates the digitized volume data to establish the volumescan 3D coordinate system.

Once the workspace coordinate framework and any volume scan coordinateframework(s) have been established, computing means 232 is furtheroperable to apply standard mathematical methods to align the scancoordinate framework(s) with the workspace framework. Knowledge of thecoordinates in both the scan framework and the workspace framework ofeach of three selected landmarks is required and is sufficient for thealignment. Such landmarks may be anatomical features, scan markers whichproduce distinctive spots in the scan, or any other feature which can beunequivocally identified in both the images made by the imaging meansand in the scan.

Using information derived from the mathematical operations used to alignthe volume scan framework with the workspace framework, computing means232 is further operable to derive transformation functions forconverting scan location coordinates describing the position of aselected point in terms of the scan framework, to workspace locationcoordinates which describe the position of the same selected point interms of the workspace framework. A term used in the art for thisconversion process, which will also be used for purposes of thisapplication, is "mapping" of coordinates from one framework to another.

Computing means 232 may also perform the converse operation, e.g. to mapcoordinates of a selected point from the workspace framework to thevolume scan framework.

In a further embodiment, the system includes means for attaching atleast two fiducial marks to instrument(s) to be used in the workspace.Alternatively, a set of instruments having at least two fiducial marksmay be provided as part of the system. These fiducial marks permittracking of the position of an operative portion of the instrument andextrapolation of its path. These operations will be described in greaterdetail hereinafter. In still another embodiment, features normallypresent on a medical instrument may be used as the fiducial marks,provided the distance between at least one of such marks and theoperative portion is measured and provided to computing means 232.

In the working embodiment depicted in FIG. 2, which is a currentlypreferred embodiment, computing means 232, digitizing means 230, anddisplay unit 240 take the form of a computer workstation of the typecommercially available, having standard image processing capability anda high-resolution monitor. In this embodiment, the digitizing of all ofthe images made by the workspace imaging means, digitizing of the volumescan data, establishment of the workspace coordinate framework, andother functions described herein for computing means 232, may beaccomplished in large part or entirely by appropriate software meansstored in the memory portion of the computer workstation.

When an image of the fiducial structure is taken by an optical imagingmeans such as a video camera or X-ray machine, a two dimensionalprojection of the structure is produced. If two such images (an imagepair) are taken are different angles, for example by cameras 200, 202 inFIG. 2, the two 2D projections can be used to reconstruct the threedimensional coordinate system of the fiducial structure, using anysuitable photogrammetric projection algorithm.

In the working embodiment, the photogrammetric projection computationsare based upon a well-known projection algorithm (Bopp, H., Krauss, H.,An orientation and calibration method for non-topographic applications,Photogrammetric Engineering and Remote Sensing, Vol. 33, Nr. 9,September 1978, pp. 1191-1196), which has previously been applied byderive from X-ray images a coordinate system referenced to a BRW-typering localizer (Siddon, R., and Barth, N., "Stereotaxic localization ofintracranial targets", Int. J. Radiat. Oncol. Biol. Phys. 13:1241-1246,1987; P. Suetens et al., "A global 3D image of the blood vessels, tumor,and simulated electrode", Acta Neurochir. 33:225-232, 1984; D.Vandermeulen et al., "A new software package for the microcomputer basedBRW stereotactic system: integrated stereoscopic views of CT data andangiograms", SPIE 593:106-114, 1985).

It should be noted that while the fiducial structure, the method and thecomputations are described primarily with reference to a Cartesiancoordinate system, other types of 3D coordinate systems may be usedinstead. Such alternate coordinate systems include sphericalcoordinates, cylindrical coordinates, and others. Any of these alternatecoordinate systems could be applied in place of the Cartesian system,with appropriate changes in the projection computations, to accomplishessentially the same goals in substantially the same way. The fiducialstructure would be used in essentially the same way. However, dependingon the type of coordinate system employed, other arrangements of thefiducial points of the fiducial structure may be desirable. For example,with a spherical coordinate system, fiducial points presented as aspheroidal array instead of a box-like array, might be more convenientfor the computations. Also, the minimum or sufficient number of fiducialpoints required for the projection computation may differ for differentprojection algorithms. The number of required fiducial points would beevident from the projection algorithm selected.

To utilize the projection technique of Bopp and Krauss in a Cartesiansystem, the fiducial structure should meet the following criteria.First, the fiducial structure must have at least six fiducial pointsarranged to define two distinct planes. Second, the actual coordinatesof each of the individual fiducial points must be known and must befixed relative to the other fiducial points. Optionally but highlydesirably, the linear distance between at least one pair of fiducialpoints should be measured from the fiducial structure and stored in thecomputing means, to provide a distance reference to calibrate theworkspace in terms of real distance units. However, other methods ofcalibrating the workspace in distance units could be used.

In the embodiment of FIG. 3, fiducial structure 300 has four rods 360each having respective upper and lower ends 310, 320. Eight fiducialpoints 361, 362, 363, 364, 365, 366, 367, 368 are formed as balls onrods 360. Each of rods 360 is fixed at its lower end 310 to a plate 314.The attachment of rods 360 to plate 314 may be either detachable orpermanent.

In the illustrated embodiment, planes 370, 372 are shown as beingparallel; this is not required, but the planes cannot be orthogonal toeach other. It is believed that the greatest accuracy in themathematical calculations will be achieved if the planes are parallel ornear to parallel.

An arrangement of the fiducial points in parallel planes and along linesperpendicular to those planes to form an open square or rectangular boxprovides a simple configuration for defining the coordinates of thefiducial points within the coordinate framework of the calibration unit.However, the trapezoidal arrangement depicted in FIG. 3 is currentlypreferred. In use, the trapezoidal fiducial structure of FIG. 3 isplaced with fiducial points 364, 365, 366 rearward and closer to thepatient, and fiducial points 361, 362, 363 forward and nearer to theworkspace imaging means. The arrangement having the "rearward" fiducialpoints of the fiducial structure spread farther apart than the forwardpoints is believed to be easier to position such that none of thefiducial points is obscured or blocked in either of the images made bycameras 200, 202. In a further preferred embodiment, the "rearward"fiducial points are constructed to be distinguishable from the "forward"fiducial points. This may be accomplished by making them of differingshapes (say boxes vs. balls), differing colors, etc.

The connecting elements constituted by rods 360 of FIG. 3 need not bearranged as a trapezoid, a rectangle or any other regular figure. Nor isit required that the fiducial points in the first plane be positioneddirectly above the points in the second plane. It will also be apparentthat the fiducial structure need not have a plate such as plate 314,rods such as rods 360, or fiducial points shaped as balls as in FIG. 3.All that is required is a minimum of six fiducial points arranged tosatisfy the conditions described in the preceding paragraphs, and meansfor holding the fiducial points in fixed relation to each other. Arather different construction might for example be a clear plasticbox-like structure with fiducial elements, either brightly visible marksor shapes such as balls, at appropriate corners. The fiducialidentifiers need not be balls as shown in FIG. 3, but could be othershapes, including pyramids or boxes; markings on rods such as rods 360;vertices at the interconnections of rod-like elements, etc.

Optionally but desirably, as in the embodiment of FIG. 3, there are twoadditional fiducial points 367, 368, beyond the six required for thecomputations. The "extra" fiducial points may be used to verify that thecomputation of locational coordinates from the camera images is correct.

FIGS. 4A and 4B depict one embodiment of a fiducial structure as itwould be seen in video and/or CRT displays of the respective images asseen by cameras 200, 202, of a workspace including a patient's head. Thepatient's head 400 has an exposed portion of the brain 402 which servesas the point of entry for a surgical procedure, and a fiducial structure404 positioned adjacent thereto.

In a still further embodiment, a grid representing the workspacecoordinate framework may be projected onto the workspace by means of alight projector analogous to a common slide projector, but using moreconcentrated light. Still another embodiment includes a spot projectorlike a laser spot, which projects a bright or colored spot onto thesurface of the patient, the spot being detectable in image pairs made bythe camera(s), and accordingly localizable by the same means as anyother selected feature in the workspace. This spot projector can beaimed by a user to select a spot whose workspace coordinates it isdesired to determine, or automatically by the computing means toindicate the coordinate location of a feature selected from another scansuch as a volume scan.

The apparatus so designed is also functional to convert from 3Dcoordinate framework established from two video 2D images, to a second3D coordinate framework established from a similar pair of X-ray 2Dimages made with a calibration unit that has radio-opaque fiducialpoints. These X-ray images could be standard-type radiological X-rays,or angiograms. This X-ray coordinate framework can further be alignedwith a volume scan framework in the same manner as for the videoframework, and location coordinates of features in the X-ray imagestransformed to video coordinates or volume scan coordinates, as desired.

A sequence of steps of a method of localizing and guiding surgicalinstruments is described as follows, referring as needed to FIG. 2. Thefirst step is to position cameras 200, 202 for viewing a medicalworkspace 208. The angle 212 between the sightlines 204, 206 ispreferably from about 30 degrees to about 150 degrees.

Next, the fiducial structure is positioned within the workspace so as tohave at least six fiducial points visible to both cameras. A pair ofimages is made in effect simultaneously, of the workspace with thefiducial structure therein, to produce a calibration image pair. Theimages from the respective cameras are digitized and 2 dimensionalcoordinates for each of the fiducial points in the 2D images made byeach of cameras 200, 202 are determined. A projection algorithm, whichin a working embodiment of the method is the Bopp-Krauss projectionalgorithm previously referenced herein, is then used to mathematicallyreconstruct a workspace 3D coordinate framework from the 2D coordinatesfrom both images of the calibration image pair, plus the known locationparameters of the fiducial points in the fiducial structure. Theprojection algorithm is optimized using a least-squares approach. All ofthe foregoing computations, and those described later, may desirably beperformed by operating a computer workstation configured similarly tocomputing means 232.

Generally, it is preferred to make the calibration workspace image pairwith the fiducial structure and the patient in the workspace, because itis easier to ensure that the desired body region of the patient isadequately centered within the workspace defined by the edges of thecamera views. However, it is not required that the patient be in theworkspace when the calibration image pair is made.

The fiducial structure 220 may be removed from the medical workspace 208at any time after the calibration image pair has been made, so long asall subsequent image pairs are made from the same two locations.

Scan data from one or more volume scans in a corresponding scan 3Dcoordinate framework are then provided to the computer. These scancoordinates may be previously stored in a memory unit within, oroperably associated with, the computer, or may be supplied at this timethrough an external data input. The workspace coordinates and scancoordinates of at least three points which can be identified in both theworkspace 3D framework and in the scan 3D framework are obtained and areused to make the alignment computations. These three points may beportions of the scan markers used in the internal scans which are alsovisible to both cameras in the workspace. Alternatively, anatomicalfeatures of the patient which can be pinpointed on both the visualimages and the volume scans may be used.

The computations for alignment of the two frameworks and transformationof coordinates from one framework to the other use a linear algebraapproach as described in theory and algorithmic solution in standardmathematical texts. Following alignment of the volume scan frameworkwith the workspace framework, coordinates in the workspace framework aredetermined for one or more medical target(s) in the workspace.

Referring to FIG. 6, a medical instrument 600 to be used in theprocedure is provided with at least two fiducial marks 610 which arevisible to both cameras during the procedure. The physical distancebetween at least one of the instrument fiducial marks 610 and thesignificant or operative portion(s) of the instrument whose position itis desired to monitor, must be known. Such an operative portion 620might, for example, be a surgical instrument (such as the cutting tip ofa scalpel), pointer, electrodes, or a tip of a medical probe. In thenext step of the method, the locational coordinates in the workspaceframework of the instrument fiducial marks 610 are determined. Fromthese coordinates and the known physical distance between one of theinstrument fiducial marks 620 and the tip 630 of the instrument, thecoordinates of the location of the instrument tip 630 are determined.The location of the instrument tip 630 relative to the location of thetarget is thereby established.

The position of the instrument including the tip, relative to the targetand other structures within the workspace is then displayed for viewingby the person guiding the instrument. Optionally, from the lineconstituted by the two instrument fiducial marks, the path of theinstrument if it moves further along that line from its present positionis extrapolated to determine whether it will approach a desired targetwhose workspace coordinates are known.

To guide the instrument tip to the desired target, a navigationalprocedure analogous to the landing of an aircraft on a runway isperformed. The workspace coordinates of the tip are determinedrepeatedly as the instrument is moved, and the positions of theinstrument and its tip are displayed relative to the target. Asnecessary, the direction of travel of the instrument is adjusted, thepath in the new direction is extrapolated, the instrument tip is movedand its location again determined. With sufficient speed in computation,it is expected that the system will be able to provide monitoring andnavigation on a time-scale approaching or substantially reachingreal-time. Such a real-time system would be highly preferred.

Table I presents results of accuracy tests of the localization apparatusand system. The tests were performed by comparing the 3D locationcoordinates derived using the video system with three-dimensionalcoordinates obtained by physical measurement with a calibratedBrown-Roberts-Wells (BRW) arc and a mockup of a patient's head.

                  TABLE I                                                         ______________________________________                                        TEST OF VIDEO LOCALIZER DEFINED TARGET                                        COMPARED TO ACTUAL TARGET AND                                                 BRW LOCALIZER DEFINED TARGET                                                               BRW LOCALIZER VIDEO LOCALIZER                                    ACTUAL STEREO-                                                                             STEREOTACTIC  STEREOTACTIC                                       TACTIC COORDINATE                                                                          COORDINATE    COORDINATE                                         ______________________________________                                        TARGET TEST 1                                                                   AP = 92.6    AP = 91.3     AP = 92.0                                         .sup. LAT = -6.8                                                                           .sup. LAT = -6.5                                                                            .sup. LAT = -5.8                                  VERT = 14.0  VERT = 14.9   VERT = 13.2                                        TARGET TEST 2                                                                   AP = -16.0   AP = -15.7    AP = -17.8                                        .sup. LAT = 25.0                                                                           .sup. LAT = 24.2                                                                            .sup. LAT = 26.1                                  VERT = 48.4  VERT = 48.1   VERT = 48.4                                        ______________________________________                                    

As indicated by the data in Table 1, the localization results presentlyobtained with a working embodiment of the invented system are accurateto within at least about 2 millimeters of the locations determined by aconventional BRW localization system.

The system, comprising the apparatus and method for localization, mayalso be applied to localize and track features responsive to aneuron-stimulating electrode. Such a use is advantageous when thesurgeon is attempting to navigate around essential structures such asthe speech center in the brain, or to locate or confirm the location ofa lesion causing a defect in neural functioning.

A method for using the guidance system to identify a neural lesion interms of a functional deficit includes the following steps. After theworkspace coordinate framework is established and the patient's head ispositioned in the workspace and readied for the procedure, an electrodeis moved slowly or at selected intervals from one position on thesurface of the brain to another. At each position the electrode isactivated to stimulate a response. When a functional deficit in theresponse is observed, the electrode path into the brain is extrapolatedfor a sufficient distance beyond the electrode tip, to reach thesuspected length of the lesion. The extrapolation is done from at leasttwo fiducial marks associated with the electrode to define itsdirection. The extrapolated path is presumed to intersect the lesioncausing the functional deficit. Movement of the electrode is repeateduntil at least one more, and desirably two more, surface positions whichcause a similar functional deficit are found. A process similar totriangulation is used to determine the location of the lesion from thetwo or three extrapolated paths.

A process similar to the above may be used to identify a critical areasuch as a speech center which the surgeon wishes to avoid damaging. Themajor difference is that instead of extrapolating the electrode pathfrom points where a functional deficit is observed, points whereelectrode stimulation causes activation of speech in the patient areused for the extrapolation.

Still other uses for the localization apparatus and method include:identifying the position of an ultrasound probe during an ultrasoundscan of a segment of the brain (or other body region); identifying theposition of the operative portions of an endoscope, fluoroscope,operating microscope, or the like, during procedures performed with suchinstruments.

The invention has been described primarily with reference toneurosurgical procedures wherein the medical workspace is the patient'shead and brain. However, the technique may also be applied to othermedical procedures where precise localization and guidance of medicalinstruments are desirable. These include plastic surgery, particularlyof face and hands, and procedures involving the spine and spinal cordregions.

Moreover, the apparatus (including the fiducial structure) and methodare not restricted to uses in a medical or surgical arena, but mayfurther be applied to any procedure in which it is desired to correlateposition information which would be available from 2D images of aworkspace (either visual or X-ray images), with 3D position datadescribing interior and/or unseen regions of the workspace.

The invention provides numerous advantages for localization duringsurgical and other medical procedures. The invention is relativelyinexpensive to practice, since the method can be performed with acommercially available computer workstation, and/or an apparatusincluding such a workstation or even a so-called personal computer asthe computing means. No cumbersome frame is required to be attached tothe patient, as in devices of the BRW type. The system provides freehand tracking of a medical instrument during a procedure, e.g. theinstrument's position can be determined without requiring that it beattached to a reference structure such as the BRW ring or any othermechanical device.

Moreover, as long as the image pairs are made from the same respectivelocations as the calibration image pair, nearly any feature in theworkspace can be accurately localized in terms of the workspacecoordinate framework. It is desired to select new locations for makingthe image pairs, to provide a better view of portions of the workspaceor any other reason, all that need be done is to reposition the fiducialstructure in the workspace to make a new pair of calibration images. Thecomputing means then can readily compute a new workspace framework, thefiducial structure can be removed and the medical procedure continuedwith a relatively short delay. These and other advantages will beapparent to those in the medical arts.

In a further embodiment, the computing means 232 is configured to"recognize" a selection of medical or surgical instruments andappliances. This recognition is achieved by configuring computing means232 with algorithms for edge detection, color recognition or both, andby including in its nonvolatile memory data correlating the detectedshape and color patterns with those of selected instruments. When aparticular instrument is held in the workspace so as to be clear ofsignificant obstructions and an image pair is made, computing means 232then can "recognize" the instrument. Highly desirable, computing means232 further provides monitor-screen and/or voice notification of theidentity of the instrument.

Subsequently, during use of the instrument, computing means 232 tracksthe position of the instrument and of the cutting tip of other relevantportion, relative to the features in the workspace such as the patient'sbody part. This tracking is accomplished by using the edge and/or colordetection algorithms for portions of the instrument which are visible inboth images of the image pair, in combination with extrapolation of theposition and direction of portions of the instrument not visible in theimage pair. In other words, the computing means is also operable, havingonce "recognized" an instrument, to recognize certain locations on theinstrument and to extrapolate the coordinates of an unseen portion suchas a cutting tip, from the identified position of one or more firstlocations. The computing means also provides information via screenand/or voice notification, of the position of the operative portion ofthe instrument relative to that of structures of interest in theworkspace.

FIG. 5 illustrates generally the internal operation of a computing meansso configured. First, a digitized image pair made prior to theintroduction of the instrument into the workspace is compared to animage pair made with the instrument in substantially complete view, andbackground subtraction is used to remove static objects in the imagefield (step 500). Methods and algorithms for this procedure are knownfrom movie compression. Preferably, when the instrument is first broughtinto the workspace it is held clear of any obstructions so as to bereadily visible in both images in an image pair.

Next, filtering algorithms are applied to sharpen the image and enhanceobject edges (step 502). Many kinds of filtering algorithms are known inthe art: a survey of filtering methods can be found in ComputerGraphics: Principles and Practice (2nd Edition) by J. D. Foley, A. vanDam, S. K. Feiner, and J. F. Hughes, Addison-Wesley Publ., Reading,Mass. (1990).

After the image has been appropriately filtered, one or both of tworecognition protocols, one based on edge detection and one on colordetection, are applied.

In an edge detection protocol (steps 504, 506), an algorithm for edgedetection is used to define the edges of the instrument and ageometrical comparison is made to match the shape of the instrument toshapes of selected instruments stored in the memory. Once the instrumentis identified (a match is found), a series of control points on theinstrument are digitized (step 508) and its orientation and tip positionare determined in terms of coordinates in the 3D workspace (step 510).This process may be accomplished by defining the instrument with aseries of three views from different angles using a grid derived from asolid sphere or other conventional 3-dimensional shape.

If color recognition is used, it will usually be necessary to providethe instruments with colored markers. A color recognition sequence(steps 512, 514) includes a color search to match the color to colors inthe database for selected instruments, followed by use of seed pointswithin colored areas to achieve object recognition. Once an object ismatched to an instrument in the database, the remaining steps 508, 510are performed as described in the preceding paragraph.

Techniques for edge detection, geometric matching, and color recognitionprotocols are known in the art; it is not important which specifictechniques are used so long as the results are accurate and can beobtained in time approaching real time with a reasonable amount ofprocessing capacity.

In tracking a surgical procedure, the next step 520 is to repeat steps500-510 and 512-514 as long as desired, using the information onidentity and position of the instrument in each image pair as a startingpoint for analysis of the next image pair.

The surgical instruments need not have special fiduciary or other marksunless color recognition is to be used. If colored or other markers areused with the instruments, these are desirably located to be readilyvisible to both cameras in the workspace and not easily obscured by thephysician's hand, etc. In the initial recognition sequence, it may bedesirable to provide image pairs of each instrument in three differentorthogonal positions in order to fully capture its dimensions and shape.A "wire model" may then be computationally constructed to define therelative coordinates of points on the instrument.

While the computations and computational sequences have been describedwith respect to a particular working embodiment, it will be recognizedby those of skill in the arts of computing and image projection thatthere are many alternate types of computations and computationalsequences that may be used to accomplish essentially the same result inthe same or similar way. For example, algorithms for implementation of aleast squares approach are many, as problem solving techniques vary andgrow within the field of numerical analysis. Also, alternate projectioncomputation methods besides that of Bopp and Krauss referenced herein,may be applied to solve the problem of mapping from a pair of 2D imagesto a 3D spatial framework.

It will also be apparent that other configurations of the components ofthe apparatus are possible and are functional to practice the invention.It will further be apparent that the precise components of the apparatuscan be varied, without departing from the spirit and concept of theinvention. The claims alone define the scope of the invention.

What is claimed is:
 1. An apparatus for establishing workspacecoordinates for features of a medical workspace including a patient'sbody region, the apparatus comprising:a first camera and a second cameracalibrated to each other and the medical workspace and positionedrelative to the medical workspace, the first and second cameras havingrespective first and second sightlines of the medical workspace, thefirst and second sightlines being arranged at an angle and the first andsecond cameras being capable of producing workspace images of themedical workspace; and a computer in communication with the first andsecond cameras, the computer configured toestablish a workspacecoordinate framework of the medical workspace, receive informationrelating to the workspace images, receive a set of scan coordinates of ascan image in a scan coordinate framework, the scan image havingidentifiable features, align the scan coordinate framework with theworkspace coordinate framework, and determine workspace coordinates ofthe identifiable features.
 2. The apparatus of claim 1, furthercomprising a display unit in communication with the computer fordisplaying images of the medical workspace, wherein the computerproduces display signal from the workspace images and scan image andsupplies display signals to the display unit.
 3. The apparatus of claim1 further including an instrument positionable within the medicalworkspace and wherein the computer is configured to determine theworkspace coordinates of the instrument from the workspace images. 4.The apparatus of claim 3, wherein the instrument includes fiducialpoints held in fixed and known spatial relation to one another andwherein the fiducials can be identified in the workspace images and thecomputer is configured to determined the workspace coordinates of thefiducials and the instrument.
 5. The apparatus of claim 4, wherein theinstrument includes a medical probe.
 6. The apparatus of claim 4,wherein the instrument includes an ultrasound probe configured to scanthe patient's body region.
 7. The apparatus of claim 4, wherein theinstrument includes an operative portion of a fluoroscope.
 8. Theapparatus of claim 3, wherein the computer determines the workspacecoordinates of the medical instrument through color recognition.
 9. Theapparatus of claim 3, wherein the instrument includes color markers andthe computer determines the workspace coordinates of the instrumentbased on the color of markers.
 10. The apparatus of claim 3, wherein thecomputer determines the coordinates of the instrument based on geometricmatching.
 11. The apparatus of claim 1, further including an instrument,wherein the computer is further configured to track the position of theinstrument relative to the features in the workspace through recognitionof a pattern in the workspace.
 12. The apparatus of claim 11, whereinthe pattern recognized by the computer is an edge of the instrument. 13.The apparatus of claim 11, wherein the pattern recognized by thecomputer is a color of the instrument.
 14. The apparatus of claim 11,wherein the pattern recognized by the computer is the color of markersin the medical workspace.
 15. An apparatus for establishing theworkspace coordinates for features of a medical workspace including apatient's body region, the apparatus comprising:a first camera and asecond camera positionable and calibrated to the medical workspace, thefirst and second cameras configured to provide a plurality of images ofthe medical workspace; a digitizer in communication with the first andsecond cameras for digitizing each of the images to produce digitalimage signals; and a computer in communication with the digitizer toreceive the digital image signals therefrom, the computer defining aworkspace coordinate framework for the medical workspace and beingconfigured todetermine workspace coordinates in the workspace coordinateframework of any selected point which can be identified from the images,receive a set of scan coordinates for selected landmarks present in ascan made in a scan coordinate framework, correlate the scan coordinateswith the workspace coordinates of the landmarks as derived from one ofthe images, and convert the scan coordinates of the landmarks in thescan to corresponding workspace coordinates in the workspace framework.16. The apparatus of claim 15, further comprising a display unit incommunication with the computer for displaying images of the medicalworkspace, wherein the computer provides display signals to the displayunit.
 17. The apparatus of claim 15, further comprising an instrument,the instrument having fiducial points held in fixed and known spatialrelation to one another, wherein the fiducials points are selectedpoints that can be identified from the images.
 18. The apparatus ofclaim 17, wherein the instrument includes a medical probe.
 19. Theapparatus of claim 17, wherein the instrument includes an ultrasoundprobe configured to scan the patient's body region.
 20. The apparatus ofclaim 17, wherein the instrument includes an operative portion of afluoroscope.
 21. The apparatus of claim 15, further including aninstrument, wherein the computer is further configured to track theposition of the instrument relative to the features in the workspacethrough recognition of a pattern in the workspace.
 22. The apparatus ofclaim 21, wherein the pattern recognized by the computer is an edge ofthe instrument.
 23. The apparatus of claim 21, wherein the patternrecognized by the computer is a color of the instrument.
 24. Theapparatus of claim 21, wherein the medical workspace includes colormarkers and the pattern recognized by the computer is the color of themarkers.
 25. The apparatus of claim 15, wherein the computer determinesthe workspace coordinates of a feature of the medical workspace throughcolor recognition.
 26. The apparatus of claim 25, further including amedical instrument in the medical workspace, the medical instrumenthaving color markers and wherein the computer determines the workspacecoordinates of the medical instrument based on recognition of the colormarkers.
 27. The apparatus of claim 15, wherein the computer determinesthe coordinates of a feature of the medical workspace through geometricmatching.